Gradient Descent

Author
Affiliation

Minjae Park

Auburn University
MATH 2660 - Spring 2026

Published

April 17, 2026

Overview

Main idea

  • Last time we wrote linear models from balance laws.
  • Today we shift from solving equations to minimizing error.
  • We start with fitting a line to data.
  • Then we use that example to introduce gradient descent.
  • This is one natural doorway toward machine learning.

Plan

  1. Write a prediction model
  2. Measure the error with a loss function
  3. Find the best parameters
  4. Compare exact least squares and gradient descent
  5. Connect the idea to machine learning

Companion file

  • The MATLAB script for today is scripts/gradient_descent_demo.m
  • It includes:
    • noisy line-fitting data
    • the exact least-squares solution
    • a gradient descent loop
    • loss and contour plots

Fitting

A simple prediction model

  • Suppose we have data points \((x_i, y_i)\).
  • We try to fit a line \[ \hat y = b + mx. \]
  • The parameters are:
    • \(b\): intercept
    • \(m\): slope
  • Different choices of \((b,m)\) give different prediction errors.

Matrix form

  • Put the data into a matrix \[ X = \begin{bmatrix} 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_n \end{bmatrix}, \qquad w = \begin{bmatrix} b \\ m \end{bmatrix}, \qquad y = \begin{bmatrix} y_1 \\ \vdots \\ y_n \end{bmatrix}. \]
  • Then our predictions are simply \[ Xw. \]

Loss function

  • We want predictions close to the data.
  • A standard choice is the mean squared error \[ J(w) = \frac{1}{2n}\|Xw-y\|^2. \]
  • Small \(J(w)\) means the line fits the data well.
  • So the fitting problem becomes: \[ \text{find } w \text{ that minimizes } J(w). \]

Exact least-squares answer

  • For this linear model, MATLAB can solve the least-squares problem directly:

    w_star = X \ y;

  • That gives the best-fitting line.

  • This is the same least-squares story you have already seen in linear algebra.

  • But it is not the only way to think about the problem.

Gradient

Why use an iterative method?

  • For a small line-fitting problem, the exact solve is easy.
  • But in larger models:
    • the data set may be huge
    • the model may have many parameters
    • the model may no longer be linear
  • Then an iterative method becomes very attractive.
  • The basic idea is: repeatedly move in a direction that makes the loss smaller.

Gradient of the loss

  • For \[ J(w)=\frac{1}{2n}\|Xw-y\|^2, \] the gradient is \[ \nabla J(w)=\frac{1}{n}X^T(Xw-y). \]
  • The gradient tells us the direction of steepest increase.
  • So to make the loss go down, we move in the opposite direction.

Gradient descent update

\[ w_{k+1} = w_k - \alpha \nabla J(w_k) \]

  • \(w_k\): current parameter guess
  • \(\nabla J(w_k)\): slope of the loss at that point
  • \(\alpha\): learning rate or step size
  • Repeat this update many times and the parameters improve

MATLAB code for the gradient

J = @(w) (1/(2*n)) * norm(X*w - y)^2;
gradJ = @(w) (1/n) * X' * (X*w - y);
  • J(w) measures how well the current line fits.
  • gradJ(w) tells us how to change the intercept and slope.

MATLAB code for gradient descent

alpha = 0.4;
num_steps = 25;
w = [-1; -2];

for k = 1:num_steps
    w = w - alpha * gradJ(w);
end
  • Start from a rough guess.
  • Each step moves downhill.
  • If alpha is too small, progress is slow.
  • If alpha is too large, the method can oscillate or diverge.

Visuals

What we can plot

  • The data and the fitted line
  • The loss value versus iteration
  • The path of gradient descent in parameter space
  • These pictures make the method much easier to understand

Typical contour picture

  • For line fitting, the loss depends on just two parameters:
    • intercept
    • slope
  • So we can draw contour lines of \(J(b,m)\).
  • Gradient descent walks downhill across those contours toward the minimum.

Exact solve vs gradient descent

Method Good when Main idea
X \ y small or moderate linear least-squares problem direct exact least-squares solve
Gradient descent large models or iterative optimization improve the parameters step by step
  • For linear regression, both approaches can work.
  • Gradient descent becomes more important as the models get bigger.

Toward ML

Why this points toward machine learning

  • Machine learning often follows this pattern:
    1. choose a model with parameters
    2. define a loss
    3. use data to reduce the loss
  • Linear regression is one of the simplest examples.
  • Neural networks use the same broad idea, just with many more parameters.

Fast answers

  • Why move opposite the gradient?
    • Because the gradient points uphill, so the negative gradient points downhill.
  • Why not always use the exact least-squares solve?
    • Exact formulas are great for small linear problems, but gradient methods scale better to large or nonlinear models.
  • What does the learning rate do?
    • It controls the step size.
  • What if the learning rate is too big?
    • The method can overshoot and fail to settle down.
  • Is this already machine learning?
    • Yes. Linear regression trained from data is one basic machine-learning model.

Summary

  • Least squares turns fitting into a minimization problem.
  • The gradient tells us how the loss changes.
  • Gradient descent improves the parameters step by step.
  • This is a core computational idea behind modern machine learning.
  • Linear algebra is still the language underneath the whole story.